Parallelizing and deduplicating backup data

ABSTRACT

A method, a system, and a computer program product for performing a backup of data are disclosed. A grid server in a plurality of grid servers is selected for deduplicating a segment of data in a plurality of segments of data contained within a data stream. The segment of data is forwarded to the selected grid server for deduplication. A zone contained within the forwarded segment of data is deduplicated using the selected server. The deduplication is performed based on a listing of a plurality of zone stamps. Each zone stamp in the plurality of zone stamps represents a zone in a plurality of zones deduplicated by at least one server in the plurality of grid servers.

P TECHNICAL FIELD

In some implementations, the current subject matter relates to dataprocessing, and in particular, to parallelizing and deduplicating backupdata.

BACKGROUND

Many information technology (“IT”) operations and activities can bescheduled to run one or more times within some periodic cycle (daily,weekly, monthly, quarterly, etc.). One such application can be databackup. Data backups can be essential to preserving and recovery of datain the event of data loss, for example. To avoid interfering with dailyuser activities, data backups can be performed during periods of lowapplication server utilization, typically, on weeknights and onweekends. The backup job workload can be the same or different dependingon how much data needs to be protected and when. In some applications,backup jobs can be scheduled and/or configured using a commercial backupapplication, an operating system shell scripting, and/or in any othermanner.

Backup applications employ a plurality of techniques to manage datadesignated for backup. One such technique includes deduplication.Deduplication can be used to eliminate redundancy in data stream createdduring the execution of periodically executed backup tasks. In somecases, deduplication can reduce data storage capacity consumption aswell as an inter-site network bandwidth. It can do so by identifying andeliminating similar and/or identical sequences of bytes in a datastream. Deduplication can also include computation of cryptographicand/or simple hashes and/or checksums, as well as one or more forms ofdata compression (e.g., file compression, rich media data compression,delta compression, etc.).

Deduplication involves identifying similar or identical patterns ofbytes within a data stream, and replacing those bytes with fewerrepresentative bytes. By doing so, deduplicated data consumes less diskstorage capacity than data that has not been deduplicated and when thedata stream must be transmitted between two geographically separatelocations, consumes less network bandwidth. Adaptive deduplicationstrategies combine inter-file and/or intra-file discovery techniques toachieve the aforementioned goals.

Deduplication can be used to reduce the amount of primary storagecapacity that is consumed by email systems, databases and files withinfile systems. It can also be used to reduce the amount of secondarystorage capacity consumed by backup, archiving, hierarchical storagemanagement (“HSM”), document management, records management andcontinuous data protection applications. In addition, it can be used tosupport disaster recovery systems which provide secondary storage at twoor more geographically dispersed facilities to protect from the totalloss of data when one site becomes unavailable due to a site disaster orlocal system failure. In such a case, deduplication helps to reduce notonly the amount of data storage consumed, but also the amount of networkbandwidth required to transmit data between two or more facilities.

Conventional deduplication-based data storage systems perform site-widededuplication by using a single compute server that is responsible fordeduplicating all data stored on one or more simple disk storage unitsthat have no deduplication processing capability. However, thesededuplication systems typically suffer from availability issues, wherefailure/loss of a single compute server can render all data stored onthe simple disk units inaccessible to the users and/or other systems. Asthe amount of backup data increases, additional disk storage units areadded, but since they cannot assist in deduplication processing, theend-to-end backup time of these systems increases to the point where itexceeds the backup window limits of the IT department's service levelagreement.

Further, using conventional magnetic disk and/or magnetic tape and tapedrive solutions that do not support deduplication, either within/amongeach medium/drive combination, each full backup of large backup imagescan consume as much capacity as the size of the large backup image. Thiscan become time-consuming and expensive to store weeks, months and/oryears of retention backup data. Moreover, with traditional deduplicationappliances that have a single deduplication engine that acts as afront-end compute node for multiple simple disk storage units, even if avery large backup image was to be sent to the deduplication appliance,the front-end compute node may have no way to process the data inparallel since it contains a single deduplication engine. Additionally,if that deduplication appliance fails during the backup process, thebackup operation can remain in a failure state until that appliance isrepaired/replaced, thereby making this single compute node architectureinefficient.

Thus, there is a need for a deduplication system that can manage databackup activities of incoming backup data streams, and maintain aconstant backup window as the amount of data to be backed up increasesover time. Moreover, there is a need for a deduplication system that canminimize storage capacity consumption within a deduplication system'sgrid servers, reduce bandwidth (e.g., wide-area network (“WAN”)bandwidth), improve backup job completion rates, enable backup capacitythat is larger than a single grid server can process, enable fasterbackup job completion, perform automatic load-balancing, etc.

SUMMARY

In some implementations, the current subject matter relates to acomputer implemented method for performing a backup of data. The methodcan include selecting a grid server in a plurality of grid servers fordeduplicating a segment of data in a plurality of segments of datacontained within a data stream, forwarding the segment of data to theselected grid server for deduplication, and deduplicating, using theselected grid server, a zone contained within the forwarded segment ofdata based on a listing of a plurality of zone stamps, each zone stampin the plurality of zone stamps representing a zone in a plurality ofzones deduplicated by at least one server in the plurality of gridservers.

In some implementations, the current subject matter can include one ormore of the following optional features. The listing of the plurality ofzone stamps can be a listing specific to the selected grid server. Thededuplicating can include comparing, using at least one zone containedin the listing specific to the selected server, the zone containedwithin the forwarded segment of data to at least one zone stored on theselected server, and determining, using the selected grid server,whether the compared zone matches at least one zone stored on theselected server. Upon determination that the compared zone matches atleast one zone stored on the selected server, the compared zone can bededuplicated. Upon determination that the compared zone does not matchat least one zone stored on the selected server, a determination can bemade as to whether the compared zone matches at least one zone stored onanother server in the plurality of grid servers using a listing of zonestamps specific to the another server.

In some implementations, the listing in the plurality of zone stamps canbe a listing of zone stamps for all servers in the plurality of gridservers.

In some implementations, the method can further include segmenting thedata stream into the plurality of segments of data, determining amaximum zone size of a zone for deduplication by each grid server in theplurality of grid servers, and determining a ratio of the maximum zonesize to a size of each segment of data in the plurality of segment ofdata. Selection of the grid server can be based on the determined ratio.

In some implementations, the method can further include selecting aplurality of grid servers for performing the deduplicating of the zonecontained within the forwarded segment of data.

In some implementations, the method can further store the deduplicatedzone on the selected grid server. Additionally, the method can includeforwarding, by the selected grid server, the deduplicated zone toanother grid server in the plurality of grid servers upon determinationby the selected grid server that storage of the deduplicated zoneexceeds a storage capacity of the selected grid server.

In some implementations, the selection, forwarding and deduplicating canbe performed in parallel for at least a portion of segments of data inthe plurality of segments of data using at least a portion of gridservers in the plurality of grid servers.

Non-transitory computer program products (i.e., physically embodiedcomputer program products) are also described that store instructions,which when executed by one or more data processors of one or morecomputing systems, causes at least one data processor to performoperations herein. Similarly, computer systems are also described thatmay include one or more data processors and memory coupled to the one ormore data processors. The memory may temporarily or permanently storeinstructions that cause at least one processor to perform one or more ofthe operations described herein. In addition, methods can be implementedby one or more data processors either within a single computing systemor distributed among two or more computing systems. Such computingsystems can be connected and can exchange data and/or commands or otherinstructions or the like via one or more connections, including but notlimited to a connection over a network (e.g., the Internet, a wirelesswide area network, a local area network, a wide area network, a wirednetwork, or the like), via a direct connection between one or more ofthe multiple computing systems, etc.

The details of one or more variations of the subject matter describedherein are set forth in the accompanying drawings and the descriptionbelow. Other features and advantages of the subject matter describedherein will be apparent from the description and drawings, and from theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this specification, show certain aspects of the subject matterdisclosed herein and, together with the description, help explain someof the principles associated with the disclosed implementations. In thedrawings,

FIG. 1 illustrates an exemplary data deduplication system fordeduplicating a stream of data received from a variety of sources,according to some implementations of the current subject matter;

FIG. 2a illustrates an exemplary network, according to someimplementations of the current subject matter;

FIG. 2b illustrates an exemplary system for an internal logical storagestructure of a grid server, according to some implementations of thecurrent subject matter;

FIG. 3 illustrates an exemplary system for performing a backup of data,according to some implementations of the current subject matter;

FIG. 4 is a flowchart illustrating an exemplary two-phase deduplicationprocess that can be performed by the system shown in FIG. 3, accordingto some implementations of the current subject matter;

FIG. 5 illustrates an exemplary system for performing a backup of data,according to some implementations of the current subject matter;

FIG. 6 is an exemplary plot illustrating a backup process performed bythe system shown in FIG. 5, according to some implementation of thecurrent subject matter;

FIG. 7 is an exemplary time-based plot illustrating a backup processperformed by a deduplication system where all grid servers have the sameprocessing/storage capacity, according to some implementation of thecurrent subject matter;

FIG. 8 illustrates an exemplary system, according to someimplementations of the current subject matter; and

FIG. 9 illustrates an exemplary method, according to someimplementations of the current subject matter.

DETAILED DESCRIPTION

To address these and potentially other deficiencies of currentlyavailable solutions, one or more implementations of the current subjectmatter provide methods, systems, articles or manufacture, and the likethat can, among other possible advantages, provide for parallelizing anddeduplicating backup data.

In some implementations, the current subject matter can performprocessing of one or more streams of data (e.g., backup streams of dataor any other data streams). The data stream can include a plurality datazones and each zone can be associated with a zone stamp that canidentify that zone. The current subject matter can generate such zonesand zone stamps based on the analysis of the received data steam(s). Thezone stamps can be compared to one another (whether or not the zones arewithin the same data stream or not).

Deduplication can reduce data storage capacity consumption and/orinter-site network bandwidth requirements. Further, data deduplicationcan reduce an amount of data storage capacity that can be consumed byprimary, backup, archive, etc. data storage systems. Further,deduplication can be effective in eliminating redundant data from beingtransferred across cost and/or bandwidth limited inter-site networks.Deduplication methods can be executed within and/or among a logicalcollection of internetworked compute and/or data storage servers, whichcan be referred to as grid servers. Grid servers can receive incomingstream(s) of data for deduplication, store data locally, and/oroptionally replicate and store remotely for some period of time. Fromthis incoming data, zones can be created. Zones can be contiguous and/ornon-contiguous segments of the incoming data, e.g., entire files,segments of files, an aggregation of multiple files, etc. For each zonein a data stream, a zone stamp can be generated and/or computed based ona content of the zone's data bytes.

A zone stamp can be a string containing up to 64 characters (and/or anyother number of characters), which, in some exemplary implementations,can be smaller than or equal to the size of the zone it represents.Because of the smaller size of zone stamps, the entire list of zonestamps can be stored in a main memory (e.g., a high-speed memorylocation) to allow them to be quickly and/or efficiently compared toeach other in order to identify zones whose content is similar and/oridentical and/or substantially identical to one another. Such zones ofdata can then be delta compressed against each other so that the zonescan be replaced by one unaltered zone and another delta compressed zonethat can contain just the bytes that are different between the zones.

In some implementations, the current subject matter relates to adeduplication grid server network that can perform deduplication ofdata. The grid server network can include a plurality of grid servers ornodes that are communicatively coupled to one another, where each gridserver can include disk storage capacity, processing units, memorycapacity, and/or networking ports for performing deduplication methods.The servers can be communicatively coupled using any type of network(e.g., wide area network (“WAN”), local area network (“LAN”),metropolitan area network (“MAN”), internet, extranet, intranet, and/orany other type of wireless and/or wired network).

Using the grid server architecture, each grid server can execute griddeduplication methods on data that can be stored within its own server.This process can be performed by the grid server independently and/or inparallel with other grid servers in the grid server network. Further,since grid servers can be interconnected via the grid server network,they can cooperate and/or communicate with one another to performdeduplication of data across all grid servers in the deduplication gridserver network. This grid deduplication activity within and amongmultiple grid servers can provide scalable performance that can becommensurate with primary storage capacity growth.

Additionally, a loss of any grid server(s) within the network may affectthe availability of the zones that it is responsible for storing,however, the current subject matter's grid server network can providefor failover/recovery models, whereby each grid server in the networkcan be a peer within the grid server network and any services can berelocated from the failed grid server to the remaining operational gridservers.

FIG. 1 illustrates an exemplary data deduplication system 100 fordeduplicating a stream of data received from a variety of sources 109(a, b, c, d, e, f, g). The source 109 can include an archive server 109a, a database server 109 b, an email server 109 c, a file server 109 d,a backup server 109 e, a document management server 109 f, a replicationserver 109 g, as well as any other application, business object,business process, business process application, server, software,hardware, etc. The system 100 can further include deduplication grids102, 104 and networks 111, 112. The network 111 can communicativelycouple the deduplication grid 102 and source of a data stream 109 andthe network 112 can communicatively couple the deduplication grid 102and the deduplication grid 104. In some implementations, thededuplication grid 102 can be located in the same physical location asthe sources 109. Alternatively, the grid 102 can be remote from thesources 109. The grid 104 can be remotely located from the sources 109and/or grid 102. For example, the grid 104 can be a disaster recoverysite for the data received from the source 109.

The grids 102 and/or 104 can include one or more computing devices,systems, servers, hardware, software, and/or any combination of hardwareand/or software, which can be communicatively coupled with one anotherusing various wireless and/or wired connections, networks, etc. Thenetworks 111, 112 can be any wireless and/or wired networks, WAN, MAN,LAN, Internet, extranet, intranet, as well any other type of network.

In some embodiments, the deduplication grid 102 can receive datastream(s) from sources 109 and can perform an initial deduplication ofthe received data. Additionally, the grid 102 can also performreconstituting original un-deduplicated data, when requested to do so bysource(s) 109. The deduplicated data can be stored in a storagesubsystem local to the grid 102 (not shown in FIG. 1). The deduplicateddata can be sent to grid 104 and stored in a storage subsystem local tothe grid 104 (not shown in FIG. 1). For example, critical applicationdata can be stored at a local facility (e.g., as represented by the grid102) and at a geographically distant remote facility (e.g., asrepresented by the grid 104) in order to provide for a full recovery inthe event of system failure, site disaster, or any other unprecedentedcondition or event.

FIG. 2a illustrates an exemplary network 200, according to someimplementations of the current subject matter. The network 200 caninclude a plurality of network sites 202 and 210 are shown in FIG. 2a ),each having a deduplication grid containing a plurality of deduplicationgrid servers 204. The grid servers 204 within each site 202 and 210 canbe communicatively coupled using any wireless and/or wired networks,WAN, MAN, LAN, Internet, extranet, intranet, as well any other type ofnetwork 206 and/or 207. The sites 202 and 210 can be communicativelycoupled using any wireless and/or wired networks, WAN, MAN, LAN,Internet, extranet, intranet, as well any other type of network 208.

In some implementations, the current subject matter can provide amulti-stage and/or multi-level deduplication of streams of data, whichcan be received by one or more servers in the network 200. In someimplementations, the data stream that can be received can be split intoa plurality of zones that can be matched against one another in order todetermine whether or not zones are similar to one another, identical,and/or substantially similar (e.g., zones that include similar datacontent). Zones having similar, identical, and/or substantially similardata content can be deduplicated using delta compression and/or datacompression. Other zones that are not similar, identical, and/orsubstantially similar to any other zone in the received data stream canbe further processed using data compression. These size-reduced zonescan then be transmitted across network 208, which can save networkbandwidth and accelerate the time it can take to replicate all of thezones.

In some implementations, the current subject matter can perform multiplesequential operations during processing of backup data stream(s) intodeduplicated and/or replicated zones. The operations can include atleast one of the following: backup stream splitting, stamp creation,stamp re-distribution, stamp matching, grid delta compression,rebalancing, purging, and/or any other operations and/or any combinationthereof. The zones can be purged from the server grid when they are nolonger required to be accessed by any of these applications 109. In someimplementations, stream splitting, stamp creation, stamp matching, griddelta compression, rebalancing, and/or purging can be performedasynchronously to one another. This can be done to maximize utilizationof system resources. The following is a discussion of each of theseprocesses and how such processes can be performed by each grid server inthe grid independently while other processes are performed across otherservers.

In some implementations, the current subject matter system can performdata stream (e.g., backup stream) splitting and/or stamp creation inaccordance with an exemplary process described in U.S. Pat. No.8,412,848 to Therrien et al., issued Apr. 2, 2013, which is incorporatedherein by reference in its entirety. In some implementations, each gridserver in the grid server network can perform data stream splittingindependently and form zones based on the received data as well ascreate zone stamps for each formed zones. Each grid server can receive adata stream that can be specifically destined for that particular serverand/or, alternatively, grid servers can receive one data stream that canbe destined for the network and determine how to split the data streamfor further processing by each grid server. Each grid server can createa stamp table that can represent all of the zones that were createdwithin that grid server based on the data in the incoming data stream.The stamp tables can be virtual stamp tables. These stamp tables can besorted from smallest to largest zone size in order to acceleratedownstream stamp matching process. During stamp matching, zone stampswhose zone sizes are +/−P percent different in size can be consideredfor matching in order to accelerate the stamp matching process.

FIG. 2b illustrates an exemplary system 250 for an internal logicalstorage structure 250 of a grid server, according to someimplementations of the current subject matter. The structure 250 can beincluded a grid server 204 (as shown in FIG. 2a ) and can include a gridserver landing area 252 and a grid server deduplication area 254. Abackup server 109 (as shown in FIG. 1) can be communicatively coupled tothe structure 250, and in particular, to the grid server landing area252.

The backup server 109 can transmit streams of backup data 256 to thegrid server 204 (as shown in FIG. 2a ). The backup data stream 256 canbe stored in the grid server landing area 252 for the purposes of“awaiting” performance of the backup process by performing compressionand/or deduplication in parallel on incoming data stream 256. The gridserver landing area 252 can include a sufficient storage capacity tocache at least the most recent full backup and/or one or more previousbackups. The data stream 256 can be temporarily stored in the gridserver landing area 252 and can provide at least one of the followingbenefits. Data backups can be completed in a shorter amount of timebecause deduplication and/or compression do not take placesimultaneously.

Further, requests to restore data are typically directed to the mostrecent backup data. This data can be cached in the in the grid serverlanding area 252 until more backups streams cause that data to be“ejected” from grid server landing area 252 and into the grid serverdeduplication area 254. In some implementations, such requests torestore data from the grid server landing area 252 might not requirethat data is reassembled from deduplicated chunks (if the data isrestored from the data that has been deduplication/replicated). Therestored data can be residing in an uncompressed and/or undeduplicatedform in the grid server landing area 252, which can accelerate restoreoperations.

Moreover, for backup system deployments where one or more grid serversare only deployed at a primary site and not at a secondary disasterrecovery site, backup tapes can be created for storage offsite at a tapestorage facility. These tapes can be produced from the grid serverlanding area 252 as soon as a backup job is completed. With the mostrecent backup data stored in uncompressed and/or undeduplicated form inthe grid server landing area 252, the throughput of creating a tape copycan be accelerated as opposed to creating a tape copy from deduplicateddata, which can require undeduplication (or “rehydration”) of databefore writing it to tape.

Additionally, for virtual machine backups, the grid server landing area252 can provide a fast and accessible NAS storage area for performing aninstant recovery of data using the grid server 204 as a temporaryprimary storage subsystem in the event of a loss of one or more virtualmachine primary storage subsystems. This can accelerate the time that itcan take to recover from the loss of virtual machine primary storage.

In some implementations, data stored in the grid server landing area 252can be deduplicated at any time and the resulting data can be stored inthe grid server deduplication area 254 of the grid server 204. In someimplementations, the current subject matter can perform a determinationas to when to move the data stored in the grid server landing area 252to the grid server deduplication area 254.

In some implementations, the grid server 204 can perform at least one ofthe following functions: backup/ingestion, deduplication, replication,and/or other functions/tasks.

The backup/ingestion function relates to receiving data from datamanagement applications 109. Applications 109 typically send largeamounts of data (e.g., gigabytes to terabytes) into one or more gridservers 204 (shown in FIG. 2a ) for processing. The data can be sentbased on a particular schedule that can be set by a backup administratorof the application 109. Minimizing an elapsed time to complete all dailybackup jobs can be most critical to the backup administrator, sincecertain customer-facing applications may need to be paused until thebackup is complete, and end-users that access that application can bedelayed in performing their work until the backup task is complete.

Another function that can be performed by the grid server 204 can bededuplication. Deduplication tasks can be designed to receive datastream that has been ingested into the grid server 204 by a datamanagement application, and segment it into zones. Each zone can belabeled a zone stamp, which can be an identifier that can be used toidentify zones that have similar, substantially similar, substantiallyidentical, and/or identical content. During the deduplication task,collection of zone stamps associated with newly created zones from arecent ingestion data stream can be compared against zone stamps thatwere previously created in order to identify similar zones usingsimilarity of their respective stamps. When a pair of zones isidentified as having similar zone stamps, the zones can bedelta-compressed to reduce consumed data storage capacity.

Yet another function of the grid server 204 can be replication. In someimplementations, a grid server 204 within the data center 202 canoptionally have their backup data replicated across a network 208 togrid servers at another site 210. The site 210 can be referred to as adisaster recovery site. A backup administrator can define apredetermined recovery point objective (“RPO”) for each backup data setbased on, for example, criticality of the data. The RPO can be a measureof time (e.g., seconds, minutes, hours, etc.) that can represent an ageof data at the second site as compared to the age of data stored at theprimary site. Ideally, an RPO of zero seconds can allow an organizationto resume operations up to the last transaction prior to the disaster atthe primary site using grid servers at the second site. For most data,RPO can be specified in terms of hours, e.g., less than 24 hours.

In some implementations, other tasks that can be run by grid servers 204can include at least one of the following: a restore task, acopy-to-tape task, a cross replication task, a purge task, a rebalancetask, etc. In some implementations, the current subject matter candynamically manage each of these additional grid server tasks and/or anyother tasks. Further, the current subject matter can balancebackup/ingestion tasks versus deduplication and/or replication tasks. Insome implementations, the current subject matter, for the purposes ofrunning these and/or any other tasks on the grid server 204, candynamically manage grid server 204 processing, memory, networking,storage and/or any other resources to ensure that ingestion performanceis not inhibited as well as to perform as much deduplication and/orreplication as possible without reducing the ingestion rate. Ingestionperformance can be one of the most important customer-visible criteriabecause the fastest ingestion performance can correspond to the shortesttime to backup, thereby affecting end-user productivity if data backupstake too long to complete. Further, during certain periods of time,ingestion performance can drop significantly due to delays introduced bythe data management application, and/or bursty network protocols (e.g.,NFS, CIFS), and/or slow data feeds from backup clients. The currentsubject matter can perform monitoring of the ingestion rate and upondetermination that the rate is low, the grid server 204 can beinstructed to increase deduplication and/or replication activityautomatically. This can reduce recovery point objective time for datathat is replicated to the second site 210.

FIG. 3 illustrates an exemplary system 300 for performing a backup ofdata, according to some implementations of the current subject matter.The system 300 can include a deduplication system 304 including aplurality of grid servers GS-A, GS-B, GS-C, . . . GS-n 308. The gridservers 308 can be communicatively coupled using a backend network 312.The network 312 can be any type of wireless and/or wired network (e.g.,Internet, extranet, intranet, WAN, MAN, LAN, etc.). A backup image 302can be supplied to the deduplication system 304 for processing via afront-end network 310. Similarly, the front-end network 310 can be anytype of wireless and/or wired network (e.g., Internet, extranet,intranet, WAN, MAN, LAN, etc.). The backup image 302 can include aplurality of segments a, b, c, . . . m (m is an integer) 306. In someimplementations, the entire backup image 302 can be supplied to thededuplication system 304 for processing. Alternatively, separatesegments 306 can be supplied to the deduplication system 304 forprocessing. In some implementations, segments 306 can be distributedamong the grid servers 308 based on various parameters (e.g., processingcapability of a server, current load of the server, storage size of theserver, etc.). In some implementations, the backup image 304 can be avery large backup image (e.g., 100 terabyte backup image).

In some implementations, data contained in the segments 306 can beingested by grid servers 308 through the network 310 and deduplicatedwithin grid servers 308, which a grid server ingests and deduplicatesthe segment 306 that has been sent to it for processing. Further, datafrom each grid server 308 can be deduplicated among all grid servers 308through the back-end network 312.

In some implementations, each segment 306 can represent a series ofcontiguous bytes of the original backup image 304 and grid servers 308can be destinations where these segments can be directed fordeduplication processing. When a segment 306 is assigned to a gridserver 308 during a backup process, the grid server 308 can split theassigned segment 306 into one or more zones. Each such zone and itsassociated zone stamp can be created in accordance with the processdisclosed in the co-owned U.S. Pat. No. 8,412,848 to Therrien et al.,issued Apr. 2, 2013, as well as co-owned and co-pending U.S. patentapplication Ser. No. 14/633,366 to Hansen et al., filed Feb. 27, 2015and entitled “Scalable Grid Deduplication”, the disclosures of which arespecifically incorporated herein by reference in their entireties.

In some implementations, the deduplication system 304 can assign andsend each segment 306 of the backup image 302 that it receives to one ormore grid servers 308 contained within the deduplication system 304. Thededuplication system 304 can also ensure that each segment 306 as wellas other data segments (of the same or other backup images) that arereceived by it can be optimally deduplicated. The deduplication system304 can include an assignment component that can determine which gridserver 308 is to receive a particular data segment 306 and assign thatdata segment to the selected grid server 308 for deduplication. Further,the deduplication system 304 can perform management of deduplicationwithin each grid server (local deduplication) and among all grid servers306 (global deduplication). In some implementations, the local andglobal deduplication processes can be implemented as a single-phaseprocess where a single global stamp table for the entire deduplicationsystem can be used to allow each new zone of each new data segment 306to be similarity matched to another zone regardless of the grid serverit was initially directed to. In alternate implementations, the localand global deduplication process can be implemented as a two-phaseprocess. In the first phase, zones of data segments 306 can be matchedwith zones within the same grid server 308 in order to minimize trafficon the backend network 312, which can accelerate the deduplicationprocess. In the second phase, stamp tables, containing zone stamps, fromeach grid server 308 can be analyzed to identify zones that were notmatched within the same grid server 308 during the first phase so thatthese zones can be deduplicated together with zones from other gridservers.

FIG. 4 is a flowchart illustrating an exemplary two-phase deduplicationprocess 400 that can be performed by the system 300 (shown in FIG. 3),according to some implementations of the current subject matter. Theprocess 400 can begin, at 402, by receiving a backup data stream forprocess (e.g., deduplication, delta-compression, etc.). Zones within thedata streams can be identified and zone stamps for those zones can begenerated. The zone determination and zone stamp generation can beperformed in accordance with an exemplary process described in U.S. Pat.No. 8,412,848 to Therrien et al., issued Apr. 2, 2013, which isincorporated herein by reference in its entirety. The zone along withcorresponding zone stamps can be received by a particular grid serverand/or all grid servers.

At 404, zone stamp matching for each new zone that is received can beperformed within a particular grid server. The new zone can be comparedto zones that have been already processed by that grid server. In someimplementations, the new zone can be compared to zones processed byother grid servers within a deduplication system.

At 406, once similar zones are identified, delta-compression process ofsimilar, identical, and/or substantially similar zones can be performed.The delta-compressed version generated as a result of thedelta-compression process can be stored in that grid server. Thedelta-compressed versions can be stored individually and/or clusteredtogether with an anchor version. Delta-compressed version clusteringand/or anchoring can be performed in accordance with an exemplaryprocess described in U.S. patent application Ser. No. 13/961,259 toVanderSpek et al., filed Aug. 7, 2013, and entitled “Delta VersionClustering and Re-Anchoring”, which is incorporated herein by referencein its entirety.

At 408, the grid server that performed operations 402-406 (and/or thededuplication system containing that grid server) can then determinewhether all new zones have been processed locally by that grid server.If not, the process returns to 404 to continue processing newly receivedzones.

Once the determination has been made that all new zones have beenprocessed locally (i.e., at the grid server receiving the new zone), theprocessing proceeds to 410. At 410, the zone stamp matching process canbe performed against other zone stamps of zones stored in other gridservers within the deduplication system (e.g., deduplication system 304shown in FIG. 3). The deduplication system can be a single deduplicationsystem containing one or more grid servers and/or can be a collection ofcommunicatively coupled multiple deduplication systems. This can bereferred to as “remote” matching.

At 412, if matching zone stamps are found at other grid serverscontained within the deduplication system, delta-compression process ofsimilar, identical, and/or substantially similar zones can be performed.The delta-compressed versions can be stored individually and/orclustered together with an anchor version. Delta-compressed versionclustering and/or anchoring can be performed in accordance with anexemplary process described in U.S. patent application Ser. No.13/961,259 to VanderSpek et al., filed Aug. 7, 2013, and entitled “DeltaVersion Clustering and Re-Anchoring”, which is incorporated herein byreference in its entirety. The delta-compressed versions can be storedby each grid server individually, e.g., each grid server stores anindividual copy of the delta-compressed version. Alternatively, thedelta-compressed versions can be stored together with an anchor based ona storage location of the anchor. In some implementations, the anchor'sstorage location can be ascertained and the newly delta-compressedversion can be transferred to that storage location for “attachment” tothe anchor. In some implementations, the delta-compressed versionsand/or their anchors can be transferred between grid servers and/orstorage locations (e.g., local, remote, disaster recovery, and/or anyother locations, etc.) as desired and/or as required (e.g., to balancestorage, to secure data, to recover data, to retrieve data, etc.).

At 414, the process 400 can determine whether all new zones have beenprocessed remotely (i.e., new zone's zone stamp compared against zonestamps of other grid servers). If not, the process returns to 410 tocontinue processing zones. Otherwise, the process terminates.

The effectiveness of the deduplication process can be dependent on theratio of a segment size to a zone size. The segment size can be definedby a backup administrator within the backup application, by thededuplication system, and/or predetermined in any other fashion. Thesegment size can be usually determined using hard split points based ona user-specified byte count within the backup application. The segmentsize is not necessarily dependent on a content-based stream splittingmodel. Within each grid server, each segment can be split into zonesbased on a content-defined splitting model. These can be natural breakpoints within data content that can absorb insert/delete modificationswithin each segment. In some exemplary, non-limiting implementations,the segment size can be selected so that it can contain hundreds tothousands of zones to benefit from the natural content-based splitpoints of zones and only incur a hard segment split point infrequently.Further, in some implementations, it is possible that first and lastzones of a segment are less likely to deduplicate well because of thesegment's hard split points.

FIG. 5 illustrates an exemplary system 500 for performing a backup ofdata, according to some implementations of the current subject matter.System 500 is similar to system 300 shown and described in connectionwith FIG. 3. In some exemplary, non-limiting implementations, the system500 can include a deduplication system 504 having a plurality of gridservers 508, including two GS-30 grid servers, one GS-20 grid server,one GS-10 grid server, and two GS-5 grid servers. The grid servers 508can be communicatively coupled using a backend network 512. The network512 can be any type of wireless and/or wired network (e.g., Internet,extranet, intranet, WAN, MAN, LAN, etc.). A backup image 502 can besupplied to the deduplication system 504 for processing via a front-endnetwork 510. Similarly, the front-end network 510 can be any type ofwireless and/or wired network (e.g., Internet, extranet, intranet, WAN,MAN, LAN, etc.).

In some exemplary, non-limiting implementations, the backup image 502can include a plurality of segments 506, e.g., 100 segments 1, 2, 3, . .. 100. In some implementations, similar to the system 300, the entirebackup image 502 can be supplied to the deduplication system 504 forprocessing, and/or separate segments 506 can be supplied to thededuplication system 504 for processing. In some implementations,segments 506 can be distributed among the grid servers 508 based onvarious parameters (e.g., processing capability of a server, currentload of the server, storage size of the server, etc.). In someimplementations, the backup image 504 can be a very large backup image(e.g., 100 terabyte backup image), larger than the storage capacity ofany one grid server.

The deduplication system 504 can assign and send each segment 506 of thebackup image 504 that it receives to one or more grid servers 508contained within the deduplication system 504. The deduplication system504 can also ensure that each segment 506 as well as other data segments(of the same or other backup images) that are received by it can beoptimally deduplicated, as discussed above with regard to FIGS. 3-4.

In some implementations, grid servers 508 can have a varying storagecapacity and can be combined within a single grid deduplication system504. As an example, to support a 100 terabyte (“TB”) backup image, sixdifferent grid servers 508 can be used, where grid servers can have thefollowing processing/storage capacities of 30, 30, 20, 10, 5, and 5 TB,as shown in FIG. 5. During the backup process, the 100 TB database canbe hard split into one hundred 1 TB segments 506 (where segment 506 a is1 TB, segment 506 b is 1 TB, etc.), as shown in FIG. 5. Each grid servercan accept the amount of storage that it can maximally accommodate,i.e., a GS-30 grid server can accept up to 30 TB of data for processing.When a particular grid server becomes full, it can generate awarning/error message indicating that it cannot accept any further datafor processing. The backup server can continue to backup up anyremaining data in that segment by re-directing it to a different gridserver, and/or it can resend the entire segment to another grid serverthat has available capacity. In some implementations, the grid servers508 can have the same and/or different processing/storage capacity.

FIG. 6 is an exemplary plot 600 illustrating a backup process performedby the system 500 shown in FIG. 5, according to some implementation ofthe current subject matter. The plot 600 is a time-based plot indicatinghow many “time units” 610 it can take each server 508 to complete abackup of data having a certain size. Thus, each of the GS-30 gridservers can consume approximately 30 time units 610 to complete backupof 30 TB of data; the GS-20 grid server can consume approximately 20time units 610 to complete backup of 20 TB of data; the GS-10 gridserver can consume approximately 10 time units 610 to complete backup of10 TB of data; and each of the GS-5 grid servers can consumeapproximately 5 time units 610 to complete backup of 5 TB of data.

In some implementations, homogenous processing capacity of the gridservers can ensure adequate load balancing across grid servers and thus,more efficient backup. As shown in FIG. 6, the first 30 TB of 100 TBbackup data, shown as first five time units 602 (similarly shaded) canleverage available capacity in all six grid servers 508 shown in FIG. 6(i.e., (6 grid servers×5 TB of data). The remaining 20 TB of the 100 TBbackup data, shown as last ten time units 604 (similarly shaded) canonly be processed by the two largest grid servers, i.e., GS-30 servers(assuming that the performance across all grid servers 508 remains thesame throughout the backup process).

In some implementations, if all grid servers 508 were identical and/orsimilar in processing/storage capacity (e.g., all 17 TB), approximately17 time units can be consumed to complete the same 100 TB backup. Insome implementations, to improve performance of the grid servers, it maybe desirable to implement smaller grid servers that can operate inparallel as opposed to fewer large grid servers.

FIG. 7 is an exemplary time-based plot 700 illustrating a backup processperformed by a deduplication system where all grid servers have the sameprocessing/storage capacity, according to some implementation of thecurrent subject matter. This system can be desired for the purposes ofoverprovisioning of grid servers to ensure occurrence of a successfulbackup of data. The plot illustrates backup operations with regard to abackup of a 100 TB data.

As shown in FIG. 7, all servers 702 (a, b, c, d, e, f) can have the sameprocessing/storage capacity to 30 TB (i.e., all are GS-30 servers). Theplot 700 shows time units 710 that may be consumed to complete backup ofdata. All six grid servers 702 can potentially accommodate 180 TB ofdata, where each grid server 702 a-e can consume 29 time units tocomplete 30 TB of data, where the grid server 702 f was able to complete5 time units and has failed thereafter, as discussed below.

In some implementations, five 30 TB grid servers 702 can accommodate 150TB backup image and any single grid server failure can cause the backupprocess to fail due to a lack of available capacity. A sixth 30 TB gridserver can alleviate this problem and allow backup to proceed, as theloss of a single grid server (e.g., due to system error, failure,disaster, etc.) can allow the backup of the 150 TB backup to besuccessfully completed (assuming that the failed grid server has notpermanently failed and can be restored by the time a restore request isissued). As shown in FIG. 7, grid server 702 f has failed at time 704.All segments that have been directed to the failed grid server 702 f canbe accommodated by the processing/storage capacity that can be availablein the remaining grid servers 702 a-e (assuming they have not failed aswell).

In some implementations, the over-provisioning can have an additionalbenefit of reducing the backup time by allowing an extra grid server toassist in the overall completion of all backup segments. Unlikeconventional array of inexpensive disks (“RAID”) schemes, where onesystem “element” can be set aside as a hot-spare awaiting the failure ofanother element, the current subject matter's overprovisioning of gridservers can use all of the grid servers to improve performance and/orprovide resiliency. The degree of overprovisioning can be increasedbeyond a single grid server in order to be able to survive multiple gridserver failures during a single backup. The number of servers and/ortheir processing/storage capacity can be changed to accommodate any sizeof backup of data. This determination can be based on a variety offactors, including historical data, system preferences, anticipatedincrease in size of backup data, etc.

In some implementations, the current subject matter can be configured tobe implemented in a system 800, as shown in FIG. 8. The system 800 caninclude a processor 810, a memory 820, a storage device 830, and aninput/output device 840. Each of the components 810, 820, 830 and 840can be interconnected using a system bus 850. The processor 810 can beconfigured to process instructions for execution within the system 800.In some implementations, the processor 810 can be a single-threadedprocessor. In alternate implementations, the processor 810 can be amulti-threaded processor. The processor 810 can be further configured toprocess instructions stored in the memory 820 or on the storage device830, including receiving or sending information through the input/outputdevice 840. The memory 820 can store information within the system 800.In some implementations, the memory 820 can be a computer-readablemedium. In alternate implementations, the memory 820 can be a volatilememory unit. In yet some implementations, the memory 820 can be anon-volatile memory unit. The storage device 830 can be capable ofproviding mass storage for the system 800. In some implementations, thestorage device 830 can be a computer-readable medium. In alternateimplementations, the storage device 830 can be a floppy disk device, ahard disk device, an optical disk device, a tape device, non-volatilesolid state memory, or any other type of storage device. Theinput/output device 840 can be configured to provide input/outputoperations for the system 800. In some implementations, the input/outputdevice 840 can include a keyboard and/or pointing device. In alternateimplementations, the input/output device 840 can include a display unitfor displaying graphical user interfaces.

FIG. 9 illustrates an exemplary method 900 for performing a backup ofdata, according to some implementations of the current subject matter.The method 900 can be performed by the systems shown and discussed abovewith regard to FIGS. 1-8. At 902, a grid server in a plurality of gridservers can be selected for deduplicating a segment of data in aplurality of segments of data contained within a data stream. Thesegment of data can be forwarded to the selected grid server fordeduplication, at 904. At 906, a zone contained within the forwardedsegment of data can be deduplicated using the selected server. Thededuplication can be performed based on a listing of a plurality of zonestamps, where each zone stamp in the plurality of zone stamps canrepresent a zone in a plurality of zones deduplicated by at least oneserver in the plurality of grid servers.

In some implementations, the current subject matter can include one ormore of the following optional features. The listing of the plurality ofzone stamps can be a listing specific to the selected grid server. Thededuplicating can include comparing, using at least one zone containedin the listing specific to the selected server, the zone containedwithin the forwarded segment of data to at least one zone stored on theselected server, and determining, using the selected grid server,whether the compared zone matches at least one zone stored on theselected server. Upon determination that the compared zone matches atleast one zone stored on the selected server, the compared zone can bededuplicated. Upon determination that the compared zone does not matchat least one zone stored on the selected server, a determination can bemade as to whether the compared zone matches at least one zone stored onanother server in the plurality of grid servers using a listing of zonestamps specific to the another server.

In some implementations, the listing in the plurality of zone stamps canbe a listing of zone stamps for all servers in the plurality of gridservers.

In some implementations, the method 900 can further include segmentingthe data stream into the plurality of segments of data, determining amaximum zone size of a zone for deduplication by each grid server in theplurality of grid servers, and determining a ratio of the maximum zonesize to a size of each segment of data in the plurality of segment ofdata. Selection of the grid server can be based on the determined ratio.

In some implementations, the method 900 can further include selecting aplurality of grid servers for performing the deduplicating of the zonecontained within the forwarded segment of data.

In some implementations, the method 900 can further store thededuplicated zone on the selected grid server. Additionally, the method900 can include forwarding, by the selected grid server, thededuplicated zone to another grid server in the plurality of gridservers upon determination by the selected grid server that storage ofthe deduplicated zone exceeds a storage capacity of the selected gridserver.

In some implementations, the selection, forwarding and deduplicating canbe performed in parallel for at least a portion of segments of data inthe plurality of segments of data using at least a portion of gridservers in the plurality of grid servers.

The systems and methods disclosed herein can be embodied in variousforms including, for example, a data processor, such as a computer thatalso includes a database, digital electronic circuitry, firmware,software, or in combinations of them. Moreover, the above-noted featuresand other aspects and principles of the present disclosedimplementations can be implemented in various environments. Suchenvironments and related applications can be specially constructed forperforming the various processes and operations according to thedisclosed implementations or they can include a general-purpose computeror computing platform selectively activated or reconfigured by code toprovide the necessary functionality. The processes disclosed herein arenot inherently related to any particular computer, network,architecture, environment, or other apparatus, and can be implemented bya suitable combination of hardware, software, and/or firmware. Forexample, various general-purpose machines can be used with programswritten in accordance with teachings of the disclosed implementations,or it can be more convenient to construct a specialized apparatus orsystem to perform the required methods and techniques.

The systems and methods disclosed herein can be implemented as acomputer program product, i.e., a computer program tangibly embodied inan information carrier, e.g., in a machine readable storage device or ina propagated signal, for execution by, or to control the operation of,data processing apparatus, e.g., a programmable processor, a computer,or multiple computers. A computer program can be written in any form ofprogramming language, including compiled or interpreted languages, andit can be deployed in any form, including as a stand-alone program or asa module, component, subroutine, or other unit suitable for use in acomputing environment. A computer program can be deployed to be executedon one computer or on multiple computers at one site or distributedacross multiple sites and interconnected by a communication network.

As used herein, the term “user” can refer to any entity including aperson or a computer.

Although ordinal numbers such as first, second, and the like can, insome situations, relate to an order; as used in this document ordinalnumbers do not necessarily imply an order. For example, ordinal numberscan be merely used to distinguish one item from another. For example, todistinguish a first event from a second event, but need not imply anychronological ordering or a fixed reference system (such that a firstevent in one paragraph of the description can be different from a firstevent in another paragraph of the description).

The foregoing description is intended to illustrate but not to limit thescope of the invention, which is defined by the scope of the appendedclaims. Other implementations are within the scope of the followingclaims.

These computer programs, which can also be referred to programs,software, software applications, applications, components, or code,include machine instructions for a programmable processor, and can beimplemented in a high-level procedural and/or object-orientedprogramming language, and/or in assembly/machine language. As usedherein, the term “machine-readable medium” refers to any computerprogram product, apparatus and/or device, such as for example magneticdiscs, optical disks, memory, and Programmable Logic Devices (PLDs),used to provide machine instructions and/or data to a programmableprocessor, including a machine-readable medium that receives machineinstructions as a machine-readable signal. The term “machine-readablesignal” refers to any signal used to provide machine instructions and/ordata to a programmable processor. The machine-readable medium can storesuch machine instructions non-transitorily, such as for example as woulda non-transient solid state memory or a magnetic hard drive or anyequivalent storage medium. The machine-readable medium can alternativelyor additionally store such machine instructions in a transient manner,such as for example as would a processor cache or other random accessmemory associated with one or more physical processor cores.

To provide for interaction with a user, the subject matter describedherein can be implemented on a computer having a display device, such asfor example a cathode ray tube (CRT) or a liquid crystal display (LCD)monitor for displaying information to the user and a keyboard and apointing device, such as for example a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well. For example,feedback provided to the user can be any form of sensory feedback, suchas for example visual feedback, auditory feedback, or tactile feedback;and input from the user can be received in any form, including, but notlimited to, acoustic, speech, or tactile input.

The subject matter described herein can be implemented in a computingsystem that includes a back-end component, such as for example one ormore data servers, or that includes a middleware component, such as forexample one or more application servers, or that includes a front-endcomponent, such as for example one or more client computers having agraphical user interface or a Web browser through which a user caninteract with an implementation of the subject matter described herein,or any combination of such back-end, middleware, or front-endcomponents. The components of the system can be interconnected by anyform or medium of digital data communication, such as for example acommunication network. Examples of communication networks include, butare not limited to, a local area network (“LAN”), a wide area network(“WAN”), and the Internet.

The computing system can include clients and servers. A client andserver are generally, but not exclusively, remote from each other andtypically interact through a communication network. The relationship ofclient and server arises by virtue of computer programs running on therespective computers and having a client-server relationship to eachother.

The implementations set forth in the foregoing description do notrepresent all implementations consistent with the subject matterdescribed herein. Instead, they are merely some examples consistent withaspects related to the described subject matter. Although a fewvariations have been described in detail above, other modifications oradditions are possible. In particular, further features and/orvariations can be provided in addition to those set forth herein. Forexample, the implementations described above can be directed to variouscombinations and sub-combinations of the disclosed features and/orcombinations and sub-combinations of several further features disclosedabove. In addition, the logic flows depicted in the accompanying figuresand/or described herein do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. Otherimplementations can be within the scope of the following claims.

1-27. (canceled)
 28. A computer-implemented method for deduplicatingdata using a deduplication system communicatively coupled to a pluralityof grid servers, comprising: splitting a received data stream into aplurality of data segments; selecting a grid server in the plurality ofgrid servers for deduplicating a segment of data in the plurality ofsegments, each grid server in the plurality of servers having a gridserver capacity, the grid server is selected based on a comparison of asize of the segment of data to the grid server capacity and a listing ofzone stamps identifying data zones previously deduplicated by theplurality of grid servers; and deduplicating, using the selected gridserver, a data zone contained in the segment of data.
 29. The methodaccording to claim 28, wherein the grid server capacity includes atleast one of the following: a processing capability of the grid server,a current processing load of the grid server, a storage size of the gridserver, and any combination thereof
 30. The method according to claim28, wherein the selecting further comprises selecting two or more gridservers in the plurality of grid servers for deduplicating the segmentof data, wherein grid server capacity of each grid server in the two ormore grid servers is less than the size of the segment of data.
 31. Themethod according to claim 30, further comprising splitting the segmentof data into a plurality of sub-segments of data for deduplicating bythe two or more grid servers, each grid server in the two or more gridservers processing a sub-segment of data based on a size of thesub-segment and corresponding grid server capacity.
 32. The methodaccording to claim 28, further comprising replicating the deduplicateddata zone to a grid server in another plurality of grid serverscommunicatively coupled to the plurality of grid servers, wherein thegrid server in the another plurality of grid servers is determined basedon a recovery point objective time associated with the grid server inthe another plurality of grid servers corresponding to a time consumedby retrieval of the deduplicated data zone from the grid server in theanother plurality of grid servers.
 33. The method according to claim 28,wherein the deduplicating further comprises determining, using thelisting of the plurality of zone stamps, by a first grid server in theplurality of grid servers that a second grid server in the plurality ofgrid servers previously deduplicated a first zone in the plurality ofzones having a first zone stamp matching to a second zone stamp of asecond zone being processed by the first grid server, and transmitting,by the first grid server, the second zone to the second grid server fordeduplication.
 34. The method according to claim 33, wherein the listingof the plurality of zone stamps is a listing specific to the selectedgrid server.
 35. The method according to claim 34, wherein thededuplicating further comprises comparing, using at least one zonecontained in the listing specific to the selected server, the zonecontained within the segment of data to at least one zone stored on theselected server; determining, using the selected grid server, whetherthe compared zone matches at least one zone stored on the selectedserver, wherein upon determination that the compared zone matches atleast one zone stored on the selected server, deduplicating the comparedzone; upon determination that the compared zone does not match at leastone zone stored on the selected server, determining whether the comparedzone matches at least one zone stored on another server in the pluralityof grid servers using a listing of zone stamps specific to the anotherserver.
 36. The method according to claim 28, wherein the listing in theplurality of zone stamps is a listing of zone stamps for all servers inthe plurality of grid servers.
 37. The method according to claim 28,further comprising segmenting the data stream into the plurality ofsegments of data; determining a maximum zone size of a zone fordeduplication by each grid server in the plurality of grid servers; anddetermining a ratio of the maximum zone size to a size of each segmentof data in the plurality of segment of data; wherein the selecting ofthe grid server is based on the determined ratio.
 38. The methodaccording to claim 28, further comprising storing the deduplicated zoneon the selected grid server.
 39. The method according to claim 38,further comprising forwarding, by the selected grid server, thededuplicated zone to another grid server in the plurality of gridservers upon determination by the selected grid server that storage ofthe deduplicated zone exceeds a storage capacity of the selected gridserver.
 40. The method according to claim 28, wherein the splitting, theselecting and the deduplicating is performed in parallel for at least aportion of segments of data in the plurality of segments of data usingat least a portion of grid servers in the plurality of grid servers. 41.A system comprising: at least one programmable processor; and anon-transitory machine-readable medium storing instructions that, whenexecuted by the at least one programmable processor, cause the at leastone programmable processor to perform operations comprising: splitting areceived data stream into a plurality of data segments; selecting a gridserver in the plurality of grid servers for deduplicating a segment ofdata in the plurality of segments, each grid server in the plurality ofservers having a grid server capacity, the grid server is selected basedon a comparison of a size of the segment of data to the grid servercapacity and a listing of zone stamps identifying data zones previouslydeduplicated by the plurality of grid servers; and deduplicating, usingthe selected grid server, a data zone contained in the segment of data.42. The system according to claim 41, wherein the grid server capacityincludes at least one of the following: a processing capability of thegrid server, a current processing load of the grid server, a storagesize of the grid server, and any combination thereof
 43. The systemaccording to claim 41, wherein the selecting further comprises selectingtwo or more grid servers in the plurality of grid servers fordeduplicating the segment of data, wherein grid server capacity of eachgrid server in the two or more grid servers is less than the size of thesegment of data.
 44. The system according to claim 43, wherein theoperations further comprise splitting the segment of data into aplurality of sub-segments of data for deduplicating by the two or moregrid servers, each grid server in the two or more grid serversprocessing a sub-segment of data based on a size of the sub-segment andcorresponding grid server capacity.
 45. The system according to claim41, wherein the operations further comprise replicating the deduplicateddata zone to a grid server in another plurality of grid serverscommunicatively coupled to the plurality of grid servers, wherein thegrid server in the another plurality of grid servers is determined basedon a recovery point objective time associated with the grid server inthe another plurality of grid servers corresponding to a time consumedby retrieval of the deduplicated data zone from the grid server in theanother plurality of grid servers.
 46. The system according to claim 41,wherein the deduplicating further comprises determining, using thelisting of the plurality of zone stamps, by a first grid server in theplurality of grid servers that a second grid server in the plurality ofgrid servers previously deduplicated a first zone in the plurality ofzones having a first zone stamp matching to a second zone stamp of asecond zone being processed by the first grid server, and transmitting,by the first grid server, the second zone to the second grid server fordeduplication.
 47. The system according to claim 46, wherein the listingof the plurality of zone stamps is a listing specific to the selectedgrid server.
 48. The system according to claim 47, wherein thededuplicating further comprises comparing, using at least one zonecontained in the listing specific to the selected server, the zonecontained within the segment of data to at least one zone stored on theselected server; determining, using the selected grid server, whetherthe compared zone matches at least one zone stored on the selectedserver, wherein upon determination that the compared zone matches atleast one zone stored on the selected server, deduplicating the comparedzone; upon determination that the compared zone does not match at leastone zone stored on the selected server, determining whether the comparedzone matches at least one zone stored on another server in the pluralityof grid servers using a listing of zone stamps specific to the anotherserver.
 49. The system according to claim 41, wherein the listing in theplurality of zone stamps is a listing of zone stamps for all servers inthe plurality of grid servers.
 50. The system according to claim 41,wherein the operations further comprise segmenting the data stream intothe plurality of segments of data; determining a maximum zone size of azone for deduplication by each grid server in the plurality of gridservers; and determining a ratio of the maximum zone size to a size ofeach segment of data in the plurality of segment of data; wherein theselecting of the grid server is based on the determined ratio.
 51. Thesystem according to claim 41, wherein the operations further comprisestoring the deduplicated zone on the selected grid server.
 52. Themethod according to claim 38, wherein the operations further compriseforwarding, by the selected grid server, the deduplicated zone toanother grid server in the plurality of grid servers upon determinationby the selected grid server that storage of the deduplicated zoneexceeds a storage capacity of the selected grid server.
 53. The systemaccording to claim 41, wherein the splitting, the selecting and thededuplicating is performed in parallel for at least a portion ofsegments of data in the plurality of segments of data using at least aportion of grid servers in the plurality of grid servers.
 54. A computerprogram product comprising a non-transitory machine-readable mediumstoring instructions that, when executed by at least one programmableprocessor, cause the at least one programmable processor to performoperations comprising: splitting a received data stream into a pluralityof data segments; selecting a grid server in the plurality of gridservers for deduplicating a segment of data in the plurality ofsegments, each grid server in the plurality of servers having a gridserver capacity, the grid server is selected based on a comparison of asize of the segment of data to the grid server capacity and a listing ofzone stamps identifying data zones previously deduplicated by theplurality of grid servers; and deduplicating, using the selected gridserver, a data zone contained in the segment of data.
 55. The computerprogram product according to claim 54, wherein the grid server capacityincludes at least one of the following: a processing capability of thegrid server, a current processing load of the grid server, a storagesize of the grid server, and any combination thereof
 56. The computerprogram product according to claim 54, wherein the selecting furthercomprises selecting two or more grid servers in the plurality of gridservers for deduplicating the segment of data, wherein grid servercapacity of each grid server in the two or more grid servers is lessthan the size of the segment of data.
 57. The computer program productaccording to claim 56, wherein the operations further comprise splittingthe segment of data into a plurality of sub-segments of data fordeduplicating by the two or more grid servers, each grid server in thetwo or more grid servers processing a sub-segment of data based on asize of the sub-segment and corresponding grid server capacity.
 58. Thecomputer program product according to claim 54, wherein the operationsfurther comprise replicating the deduplicated data zone to a grid serverin another plurality of grid servers communicatively coupled to theplurality of grid servers, wherein the grid server in the anotherplurality of grid servers is determined based on a recovery pointobjective time associated with the grid server in the another pluralityof grid servers corresponding to a time consumed by retrieval of thededuplicated data zone from the grid server in the another plurality ofgrid servers.
 59. The computer program product according to claim 54,wherein the deduplicating further comprises determining, using thelisting of the plurality of zone stamps, by a first grid server in theplurality of grid servers that a second grid server in the plurality ofgrid servers previously deduplicated a first zone in the plurality ofzones having a first zone stamp matching to a second zone stamp of asecond zone being processed by the first grid server, and transmitting,by the first grid server, the second zone to the second grid server fordeduplication.
 60. The computer program product according to claim 59,wherein the listing of the plurality of zone stamps is a listingspecific to the selected grid server.
 61. The computer program productaccording to claim 60, wherein the deduplicating further comprisescomparing, using at least one zone contained in the listing specific tothe selected server, the zone contained within the segment of data to atleast one zone stored on the selected server; determining, using theselected grid server, whether the compared zone matches at least onezone stored on the selected server, wherein upon determination that thecompared zone matches at least one zone stored on the selected server,deduplicating the compared zone; upon determination that the comparedzone does not match at least one zone stored on the selected server,determining whether the compared zone matches at least one zone storedon another server in the plurality of grid servers using a listing ofzone stamps specific to the another server.
 62. The computer programproduct according to claim 54, wherein the listing in the plurality ofzone stamps is a listing of zone stamps for all servers in the pluralityof grid servers.
 63. The computer program product according to claim 54,wherein the operations further comprise segmenting the data stream intothe plurality of segments of data; determining a maximum zone size of azone for deduplication by each grid server in the plurality of gridservers; and determining a ratio of the maximum zone size to a size ofeach segment of data in the plurality of segment of data; wherein theselecting of the grid server is based on the determined ratio.
 64. Thecomputer program product according to claim 54, wherein the operationsfurther comprise storing the deduplicated zone on the selected gridserver.
 65. The computer program product according to claim 64, whereinthe operations further comprise forwarding, by the selected grid server,the deduplicated zone to another grid server in the plurality of gridservers upon determination by the selected grid server that storage ofthe deduplicated zone exceeds a storage capacity of the selected gridserver.
 66. The computer program product according to claim 54, whereinthe splitting, the selecting and the deduplicating is performed inparallel for at least a portion of segments of data in the plurality ofsegments of data using at least a portion of grid servers in theplurality of grid servers.